About Projects Articles

±50% Performance in 14 patches or less

Don't underestimate small gains/losses

cute baby monkey determined to optimize code.
Author: Ryan Chanlatte

Special Thanks to Nitzan Wilnai and Murilo Mesquita for review.
Published: 2024-05-27
Last updated: 2024-06-06

    It's said that a program tends to spend 90% of its time in 10% of its codebase. Even amongst that 10%, developers will hyperfixate on a small fraction, usually one or two functions/loops. This is worthwhile and useful, but you shouldn't leave smaller gains elsewhere. Those small gains can snowball into bigger ones given enough of them; however, while intuitively true, how do we reason about that? How many regressions does it take until we reach our breaking point? How much performance do we need to gain, per patch, to reach an agreed upon target?

    By reframing the problem we can tackle some of those questions. Framed in terms of a, slightly modified, annual compound interest formula we get this:

\begin{align} \Large \textcolor{white} { \displaylines { &A&=&P(1+r)^{t} \\ \Rightarrow &\Delta_{\;change\,in\,baseline}&=&(1 + d\,\%_{\,per\:patch})^{p_{\,patch\:count}} } } \end{align}

    Let's make sense of this. Our principal is the baseline of whatever metric we wish to optimize. Let's say, execution time, memory usage, etc. For simplicity's sake, let's make it one so we can ignore it for now. \(\Delta\) is the resultant change to our baseline, \(p\) is the number of patches, and \( d\,\% \) is the target percentage change per patch (as measured by your benchmarking suite of choice).

    With some trivial algebra we can rearrange our above equation and answer the questions posed earlier:

\begin{align} \Large \textcolor{white} { \displaylines { &p&=&\frac{ln(\Delta)}{ln(1+d\,\%)} \\ &d\,\%&=&\Delta^{^{\frac{1}{p}}}+1 } } \end{align}

graph displaying the slightly modified annual compound interest formula

Note the scale difference between the X-axis and Y-axis.

Using this, developers can answer a couple of useful questions:

  1. How many patches would it take to improve performance by x%?
    1. Assuming a reduction of 5% per patch (a delta percentage of -0.05), it'd take approximately 14 patches to achieve an aggregate ~50% reduction in whatever metric you are trying to optimize.
  2. How much performance would we need to gain, per patch, before we've reached our goal?
    1. Suppose we have five patches before we ship and we want to reduce runtime by 10%. We'd need to reduce runtime by 2.5% (-0.025), per patch, to accomplish this.
  3. How many performance regressions can we handle before we reach our tipping point?
    1. If we're able to tolerate a 50% increase in memory before shipping; and, four patches before we shipped we would need to ensure each patch didn't exceed a 10% memory increase to ensure our goal was met.

There are some important caveats to this modeling of the problem, namely:

    In all, despite the caveats there are advantages to this approach. This method can provide a practical framework for managing performance plans more effectively. It helps set clear, achievable goals or boundaries that teams can use to navigate optimizating a codebase. Additionally, this approach supports developers focused on performance, offering them a structured way to justify enhancements or provide concrete guardrails on development trajectories.

    Circling back to the start, don't hyperfixate on a single function or loop when optimizing your program's hot path. Avoid immediately writing off a change whose improvement may be small as it can yield big gains overtime. Further, don't rebut suggestions to improve regressions as "premature optimization" because you might have the team pay for it later.

⠀